Depth acquisition device and depth acquisition method

ABSTRACT

A depth acquisition device includes a memory and a processor. The processor performs; acquiring timing information indicating a timing at which a light source irradiates a subject with infrared light; acquiring an infrared light image stored in the memory, the infrared light image being generated by imaging a scene including the subject with the infrared light according to the timing indicated by the timing information; acquiring a visible light image stored in the memory, the visible light image being generated by imaging a substantially same scene as that of the infrared light image, with visible light from a substantially same viewpoint and at a substantially same imaging time of those of the infrared light image; detecting a dust region showing dust from the infrared light image; and estimating a depth of the dust region based on the infrared light image, the visible light image, and the dust region.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. continuation application of PCT InternationalPatent Application Number PCT/JP2019/033601 filed on Aug. 28, 2019,claiming the benefit of priority of Japanese Patent Application Number2018-174276 filed on Sep. 18, 2018, the entire contents of which arehereby incorporated by reference.

BACKGROUND 1. Technical Field

The present disclosure relates to depth acquisition devices and the likewhich acquire a depth of image.

2. Description of the Related Art

Conventionally, a distance measurer for measuring a distance to asubject of image has been proposed (for example, refer to JapaneseUnexamined Patent Application Publication No. 2011-64498 (PTL 1)). Thisdistance measurer includes a light source and an imager. The lightsource irradiates the subject with light. The imager images the lightreflected on the subject. Then, the distance measurer converts eachpixel value in the image generated by the imaging into a distance to thesubject, thereby measuring the distance to the subject. In other words,the distance measurer acquires a depth of the image generated by theimager.

SUMMARY

However, the distance measurer in PTL 1 has a problem of failing toaccurately acquire the depth.

Therefore, the present disclosure provides a depth acquisition devicecapable of accurately acquiring a depth of an image.

In accordance with an aspect of the present disclosure, a depthacquisition device includes: a memory; and a processor, wherein theprocessor performs: acquiring timing information indicating a timing atwhich a light source irradiates a subject with infrared light; acquiringan infrared light image stored in the memory, the infrared light imagebeing generated by imaging a scene including the subject with theinfrared light according to the timing indicated by the timinginformation; acquiring a visible light image stored in the memory, thevisible light image being generated by imaging a substantially samescene as the scene of the infrared light image, with visible light froma substantially same viewpoint as a viewpoint of the imaging theinfrared light image at a substantially same time as an imaging time ofimaging the infrared light image; detecting a dust region showing dustfrom the infrared light image; and estimating a depth of the dust regionbased on the infrared light image, the visible light image, and the dustregion.

It should be noted that general or specific aspects of the presentdisclosure may be implemented to a system, a method, an integratedcircuit, a computer program, a computer-readable recording medium suchas a Compact Disc-Read Only Memory (CD-ROM), or any given combinationthereof. The recording medium may be a non-transitory recording medium.

The depth acquisition device according to the present disclosure iscapable of accurately acquiring a depth of an image. Additionaladvantages and effects of the aspect of the present disclosure will beapparent from the Description and the Drawings. The advantages and/oreffects may be individually obtained by the various embodiments and thefeatures of the Description and the Drawings, which need to all beprovided in order to obtain one or more such advantages and/or effects.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the disclosure willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the present disclosure.

FIG. 1 is a block diagram illustrating a hardware structure of a depthacquisition device according to Embodiment;

FIG. 2 is a schematic diagram illustrating a pixel array in asolid-state imaging element according to Embodiment;

FIG. 3 is a time chart illustrating a light emitting timing of a lightemitting element of a light source and exposure timings of a first pixelin the solid-stage imaging element according to Embodiment;

FIG. 4 is a block diagram illustrating an example of a functionalstructure of the depth acquisition device according to Embodiment;

FIG. 5 is a block diagram illustrating another example of a functionalstructure of the depth acquisition device according to Embodiment;

FIG. 6 is a flowchart illustrating overall processing operation of thedepth acquisition device according to Embodiment;

FIG. 7 is a flowchart illustrating overall processing operation by aprocessor of the depth acquisition device according to Embodiment;

FIG. 8 is a block diagram illustrating a functional structure of theprocessor of the depth acquisition device according to Embodiment;

FIG. 9A is a diagram illustrating an example of an IR image;

FIG. 9B is a diagram illustrating an example of a BW image;

FIG. 10 is a diagram illustrating an example of a binarized image to beobtained by binarization of IR image;

FIG. 11 is a diagram illustrating an example of a dust candidate regionin an IR image;

FIG. 12 is a diagram illustrating an example of FOE detected for a BWimage;

FIG. 13 is a diagram illustrating an example of a principal axisdetected for a dust region;

FIG. 14 is a diagram illustrating examples of a dust region and anon-dust region;

FIG. 15A is a diagram illustrating another example of the IR image;

FIG. 15B is a diagram illustrating another example of the BW image;

FIG. 16 is a diagram illustrating an example of a binarized imageobtained by binarization of an IR image;

FIG. 17 is a diagram illustrating an example of a dust candidate regionin an IR image;

FIG. 18 is a diagram illustrating an example of FOE detected for a BWimage;

FIG. 19 is a diagram illustrating an example of arrangement of each dustcandidate region;

FIG. 20 is a diagram illustrating a simulation result of a depthacquisition device in Embodiment;

FIG. 21 is a flowchart illustrating an example of overall processingoperation of the depth acquisition device shown in FIG. 8;

FIG. 22 is a flowchart illustrating an example of detailed processing ofsteps S31 to S34 of FIG. 21;

FIG. 23 is a flowchart illustrating another example of detailedprocessing of steps S31 to S34 of FIG. 21.

FIG. 24 is a flowchart illustrating an example of alternative processingof steps S31 to S34 of FIG. 21;

FIG. 25 is a flowchart illustrating another example of detailedprocessing of steps S31 to S34 of FIG. 21;

FIG. 26 is a block diagram illustrating an example of functionalstructure of a depth acquisition device in a variation of Embodiment;and

FIG. 27 is a block diagram illustrating another example of functionalstructure of a depth acquisition device in a variation of Embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENT

(Findings on Which the Present Disclosure is Based)

The present inventors have found that the following problems occur inconnection to the distance measurer of PTL 1 described in the“BACKGROUND ART” section.

The distance measurer of PTL 1, as described above, acquires an image byirradiating light from a light source to a subject and imaging thesubject irradiated with light, and measures depth of the image. In themeasurement of depth, Time Of Flight (TOF) is used. In such a distancemeasurer, imaging at mutually different imaging conditions is performedto improve distance measurement accuracy. That is, the distance measurerperforms imaging according to a predetermined imaging condition, anddepending on that imaging result, sets an imaging condition differentfrom the predetermined imaging condition. Then, the distance measurerperforms imaging again according to the set imaging condition.

However, in an image obtained by the imaging, dust which is present nearthe distance measurer may be reflected as noise. Consequently, it is notpossible to remove noise which is the dust only from the image on whichthe dust is projected, and it is not possible to measure depthcorrectly. Even if the imaging conditions are changed, it may bedifficult to easily suppress the reflection of dust. Further, forexample, if a distance measurer mounted on a vehicle repeats imagingunder different imaging conditions while the vehicle is traveling, sincethe viewpoint position of the repeated imaging will differ, each sceneof a plurality of images to be obtained will be different. That is, itis not possible to repeat imaging for the same scene, and thus it is notpossible to appropriately estimate the depth of the image on which thescene is projected, particularly the depth of the region on which dustis projected.

In order to solve the above problem, in accordance with an aspect of thepresent disclosure, a depth acquisition device includes: a memory; and aprocessor, wherein the processor performs: acquiring timing informationindicating a timing at which a light source irradiates a subject withinfrared light; acquiring an infrared light image stored in the memory,the infrared light image being generated by imaging a scene includingthe subject with the infrared light according to the timing indicated bythe timing information; acquiring a visible light image stored in thememory, the visible light image being generated by imaging asubstantially same scene as the scene of the infrared light image, withvisible light from a substantially same viewpoint as a viewpoint of theimaging the infrared light image at a substantially same time as animaging time of imaging the infrared light image; detecting a dustregion showing dust from the infrared light image; and estimating adepth of the dust region based on the infrared light image, the visiblelight image, and the dust region.

Since this allows the dust region to be detected from the infrared lightimage and, in that dust region, depth is estimated based on not only theinfrared light image but also the visible light image, it is possible toappropriately acquire the depth of the dust region. That is, the sceneto be imaged is substantially the same between the infrared light imageand the visible light image, and the viewpoint and the imaging time arealso substantially the same. Here, one example of images ofsubstantially the same scene, which are imaged at substantially sameviewpoint and imaging time, are images which are imaged by differentpixels of the same imaging element. These images are similar to eachchannel image of red, green, and blue of a color image, which is imagedwith a color filter of Bayer array, in which the viewing angle, theviewpoint point, and the imaging time of each image are substantiallyequal. That is, images of a substantially same scene, which are imagedat substantially same viewpoint and the imaging time, will not differ inthe position on the image of a subject by not less than two pixels ineach of imaged images. For example, when a point light source havingvisual light and infrared components is present in a scene, and only onepixel is imaged to have higher luminance in the visible light image, thepoint light source is imaged in the infrared light image as well withinthe distance closer than two pixels from the pixel corresponding to theposition of the pixel imaged in the visible light image. Moreover, thesubstantially same imaging time indicates that the imaging time is equalwithin a difference of one frame or less. Therefore, the infrared lightimage and the visible light image have high correlation with each other.However, when there is dust near the camera that performs imaging of aninfrared light image and a visible light image, irradiated infraredlight is strongly reflected on the dust, and the dust may be reflectedin the infrared light image due to the strong reflected light.Therefore, it is likely that even if dust is reflected in the infraredlight image, dust will not be reflected in the visible light image.Therefore, information lacking in the dust region can be supplementedfrom a region in the visible light image corresponding to the dustregion (that is, a corresponding region). Consequently, it is possibleto eliminate the effect of noise, which is dust, thereby appropriatelyacquiring the depth of the dust region.

For example, it is possible that in the estimating of the depth of thedust region, the processor performs: estimating first depth informationindicating a depth at each position in the infrared light image;estimating second depth information indicating a corrected depth at eachposition in the dust region, the corrected depth being obtained bycorrecting, based on the visible light image, a depth at each positionin the dust region which is indicated in the first depth information;and generating third depth information indicating (i) a depth at eachposition in a region other than the dust region in the infrared lightimage which is indicated in the first depth information and (ii) a depthat each position in the dust region in the infrared light image which isindicated in the second depth information. It is noted that in theestimating of the first depth information, TOF, etc. may be applied tothe infrared light image.

As a result of this, the third depth information indicates a depthobtained from the infrared light image as the depth of a region otherthan the dust region in the infrared light image, and a depth obtainedfrom the infrared light image and corrected based on the visible lightimage as the depth of the dust region of the infrared light image.Therefore, even in a case where there is a dust region in an infraredlight image, it is possible to appropriately estimate the depth of theentire infrared light image.

It is also possible that in the detecting of the dust region, theprocessor performs detecting, as the dust region, a higher luminanceregion having a luminance not less than a first threshold in theinfrared light image, when the higher luminance region satisfies a firstcondition.

The luminance tends to be higher within a dust region. Therefore, it ispossible to narrow down a potential region on which dust is likely to beprojected, by detecting a higher luminance region having a luminance notless than a first threshold in the infrared light image. Further, sincethe higher luminance region satisfying the first condition is detectedas the dust region, it is possible to detect the dust region with higheraccuracy by appropriately setting the first condition.

It is further possible that the first condition is that a center ofgravity of the higher luminance region is located on a straight or anarc, the straight line or the arc intersecting (i) a center of gravityof each of at least two higher luminance regions other than the higherluminance region in the infrared light image and (ii) a Focus ofExpansion (FOE) of one of the infrared light image and the visible lightimage.

For example, when imaging one dust, if the camera performs multipleexposures for the imaging of one frame of infrared light image, the dustis projected on the infrared light image as a higher luminance regionevery time that exposure is performed. Accordingly, when theabove-described camera is mounted on a moving body such as a vehicle andis moving, it is highly likely that the dust is projected on theinfrared light image as if it were blown out from an FOE. For example,if the moving speed of the above-described moving body is high, thesemultiple higher luminance regions attributable to dust tend to bearranged on a straight line that intersects the FOE. Alternatively, ifthe distortion of the lens of the camera is large, multiple higherluminance regions attributable to dust tends to be arranged on an arcthat intersects the FOE. Therefore, by setting the first condition thatthe centers of gravity of at least three higher luminance regions arearranged on the above-described straight line or arc, it is madepossible to detect a dust region with high accuracy.

It is noted that since the scene to be imaged is substantially the sameand the viewpoint is also substantially the same between in the infraredlight image and in the visible light image, the FOE of the infraredlight image and the FOE of the visible light image are substantially thesame.

It is still further possible that the first condition is that one of aprinciple axis of the higher luminance region and a line extending fromthe principle axis intersects a Focus of Expansion (FOE) of one of theinfrared light image and the visible light image.

For example, contrary to the above-described example, if the movingspeed of the moving body on which the camera is mounted is low, theabove-described plurality of higher luminance regions overlap with eachother. Consequently, these higher luminance regions are projected on theinfrared light image as one higher luminance region having an elongatedshape, in other words, a tailed shape. The principal axis or extensionof the principal axis of such a higher luminance region attributable todust tends to intersect an FOE. Therefore, by setting the firstcondition that the principal axis or extension of the principal axis ofsuch higher luminance region intersect the FOE, it is made possible todetect the dust region with higher accuracy.

It is still further possible that in the detecting of the dust region,the processor performs detecting the higher luminance region as the dustregion, when the higher luminance region further satisfies a secondcondition.

For example, a property that even if a dust region is observed in aninfrared light image, it is not observed in a visible light image may beused as the second condition. Consequently, it is possible to detect adust region with higher accuracy.

It is still further possible that the second condition is that aluminance of a position in the visible light image is less than a secondthreshold, the position in the visible light image corresponding to acenter of gravity of the higher luminance region in the infrared lightimage.

Since a dust region is not observed in the BW image, the luminance ofthe position in a visible light image corresponding to the center ofgravity of the dust region tends to be lower. Therefore, by setting thesecond condition that the luminance of the position in the visible lightimage corresponding to the center of gravity of the higher luminanceregion in the infrared light image is less than a second threshold, itbecomes possible to detect the dust region with higher accuracy.

It is still further possible that the second condition is that acorrelation coefficient between (i) a luminance in the higher luminanceregion in the infrared light image and (ii) a luminance in a region inthe visible light image is less than a third threshold, the region inthe visible light image corresponding to the higher luminance region.

Since there is a property that even if a dust region is observed in theinfrared light image, it is not observed in the visible light image,correlation between the luminance in a dust region of the infrared lightimage and the luminance in the region of the visible light imagecorresponding to the dust region tends to become lower. Therefore, bysetting the second condition that the correlation coefficient betweenthe luminance in a higher luminance region of the infrared light imageand the luminance in the region of the visible light image correspondingto the higher luminance region is less than a third threshold, it ispossible to detect the dust region with higher accuracy.

It is still further possible that in the estimating of the depth of thedust region, the processor performs: estimating depth informationindicating a depth at each position in the infrared light image; andcorrecting a depth at each position in the dust region which isindicated in the depth information, by inputting the infrared lightimage, the visible light image, the dust region, and the depthinformation into a learning model.

If a learning model is trained in advance to output a correct depth ateach position in the dust region upon input of an infrared light image,a visible light image, a dust region, and depth information, it ispossible to appropriately correct the depth information estimated fromthe infrared light image. That is, it is possible to appropriatelycorrect the depth at each position in the dust region indicated by thedepth information.

In accordance with another aspect of the present disclosure, a depthacquisition device includes: a memory; and a processor, wherein theprocessor performs: acquiring timing information indicating a timing atwhich a light source irradiates a subject with infrared light; acquiringan infrared light image stored in the memory, the infrared light imagebeing generated by imaging a scene including the subject with theinfrared light according to the timing indicated by the timinginformation; acquiring a visible light image stored in the memory, thevisible light image being generated by imaging a substantially samescene as the scene of the infrared light image, with visible light froma substantially same viewpoint as a viewpoint of the imaging theinfrared light image at a substantially same time as an imaging time ofimaging the infrared light image; estimating depth informationindicating a depth at each position in the infrared light image; andcorrecting a depth at each position in a dust region showing dust in theinfrared light image by inputting the infrared light image, the visiblelight image, and the depth information into a learning model, the depthat each position in the dust region being indicated in the depthinformation.

If a learning model is trained in advance to output a correct depth ateach position in the dust region of the infrared light image upon inputof the infrared light image, the visible light image, and the depthinformation, it is possible to appropriately correct the depthinformation estimated from the infrared light image. That is, it ispossible to appropriately correct the depth at each position in the dustregion indicated by the depth information without detecting the dustregion.

In accordance with still another aspect of the present disclosure, adepth acquisition device includes: a memory; and a processor, whereinthe processor performs: acquiring an infrared light image stored in thememory, the infrared light image being generated by imaging withinfrared light; acquiring a visible light image stored in the memory,the visible light image being generated by imaging with visible lightfrom a substantially same viewpoint as a viewpoint of the imaging theinfrared light image at a substantially same time as an imaging time ofimaging the infrared light image; detecting, as a dust region, a regionshowing dust from the infrared light image; and estimating a depth ofthe dust region based on the visible light image.

This makes it possible to eliminate the effect of noise which is dust,and appropriately acquire a depth of the dust region as in the depthacquisition device according to the above-described one aspect of thepresent disclosure.

It should be noted that general or specific aspects of the presentdisclosure may be implemented to a system, a method, an integratedcircuit, a computer program, a computer-readable recording medium suchas a Compact Disc-Read Only Memory (CD-ROM), or any given combinationthereof. The recording medium may be a non-transitory recording medium.

Hereinafter, an embodiment will be described in detail with reference tothe accompanying Drawings.

It should be noted that the following embodiment is a general orspecific example of the present disclosure. The numerical values,shapes, materials, elements, arrangement and connection configuration ofthe elements, steps, the order of the steps, etc., described in thefollowing embodiment are merely examples, and are not intended to limitthe present disclosure.

It should also be noted that the respective figures are schematicdiagrams and are not necessarily precise illustrations. Additionally,components that are essentially the same share like reference signs inthe figures.

Embodiment

[Hardware Configuration]

FIG. 1 is a block diagram illustrating a hardware configuration of depthacquisition device 1 according to Embodiment. Depth acquisition device 1according to the present embodiment has a hardware configuration whichis capable of acquiring an image based on infrared light (or nearinfrared light) and an image based on visible light by imaging of asubstantially same scene, the imaging being performed at a substantiallysame viewpoint and imaging time. It should be noted that substantiallysame means “the same to the extent that the effects in the presentdisclosure can be achieved.”

As shown in FIG. 1, depth acquisition device 1 is configured to includelight source 10, solid-state imaging element 20, processing circuit 30,diffusion plate 50, lens 60, and band-pass filter 70.

Light source 10 irradiates irradiation light. More specifically, lightsource 10 emits irradiation light to be irradiated to a subject at atiming indicated by a light emission signal generated in processingcircuit 30.

Light source 10 is configured to include, for example, a capacitor, adriving circuit, and a light emitting element, and emits light bydriving the light emitting element with electric energy accumulated inthe capacitor. The light emitting element is implemented by, as anexample, a laser diode, a light emitting diode, and the like. It shouldbe noted that light source 10 may be configured to include one kind oflight emitting element, or configured to include plural kinds of lightemitting elements according to purposes.

Hereinafter, the light emitting element is, for example, a laser diodethat emits near infrared light, or a light emitting diode that emitsnear infrared light, or the like. However, the irradiation lightirradiated by light source 10 may be infrared light (also referred to asinfrared ray) of a frequency band other than near infrared light.Hereinafter, in the present embodiment, although the irradiation lightirradiated by light source 10 will be described as infrared light, theinfrared light may be near infrared light, or infrared light of afrequency band other than that of near infrared light.

Solid-state imaging element 20 images a subject and outputs an imagingsignal indicating an exposure amount. To be more specifically,solid-state imaging element 20 performs exposure at a timing indicatedby an exposure signal generated in processing circuit 30, and outputs animaging signal indicating an exposure amount.

Solid-state imaging element 20 has a pixel array in which a first pixelthat performs imaging with reflected light, which is irradiation lightreflected by a subject, and a second pixel that images the subject aredisposed in an array. Solid-state imaging element 20 may have, forexample, as needed, cover glass, and a logic function such as an A/Dconverter, etc.

Hereinafter, as with the irradiation light, description will be madesupposing that the reflected light is infrared light. However, thereflected light does not need to be limited to infrared light providedthat the light is irradiation light reflected by a subject.

FIG. 2 is a schematic diagram illustrating pixel array 2 included insolid-state imaging element 20.

As shown in FIG. 2, pixel array 2 is configured to be disposed in anarray pattern such that first pixel 21 (IR pixel) that performs imagingwith reflected light, which is irradiation light reflected by a subject,and second pixel 22 (BW pixel) that images the subject are alternatelyaligned in columns.

Moreover, in FIG. 2, although second pixel 22 and first pixel 21 arearranged to be adjacent to each other in the row direction and aredisposed to be aligned in a stripe pattern in the row direction, inpixel array 2, this is not limiting and they may be disposed everymultiple rows (every two rows, for example). That is, the first row inwhich second pixels 22 are arranged to be adjacent to each other in therow direction, and the second row in which first pixels 21 are arrangedto be adjacent to each other in the row direction may be disposedalternately every M rows (M is a natural number). Further, the firstrow, in which second pixels 22 are arranged to be adjacent to each otherin the row direction, and the second row, in which first pixels 21 arearranged to be adjacent to each other in the row direction, may bedisposed every different number of rows (N rows of the first row and Lrows of the second row are alternately repeated (N and L are differentnatural numbers)).

First pixel 21 is implemented by, for example, an infrared light pixelsensitive to infrared light which is the reflected light. Second pixel22 is implemented by, for example, a visible light pixel sensitive tovisible light.

The infrared light pixel is configured to include, for example, anoptical filter (also called as an IR filter) which transmits onlyinfrared light, a micro lens, a light receiving element as aphotoelectric converter, and an accumulator that accumulates electriccharge generated at the light receiving element. Therefore, an imageindicating the luminance of infrared light is represented by an imagingsignal outputted from a plurality of infrared light pixels (that is,first pixel 21) included in pixel array 2. Hereinafter, this image ofinfrared light is also referred to as IR image or infrared image.

Moreover, the visible light element is configured to include, forexample, an optical filter (also called as a BW filter) which transmitsonly visible light, a micro lens, a light receiving element as aphotoelectric converter, and an accumulator that accumulates electriccharge converted at the light receiving element. Therefore, the visiblelight pixel, that is, second pixel 22, outputs an imaging signalindicating luminance and color difference. That is, a color image thatindicates luminance and color difference of visible light is representedby an imaging signal outputted from a plurality of second pixels 22included in pixel array 2. It should be noted that the optical filter ofvisible light pixel may transmit both visible light and infrared light,or may transmit only light of a specific wavelength such as red (R),green (G), or blue (B) of visible light.

Moreover, the visible light pixel may detect only the luminance ofvisible light. In this case, the visible light pixel, that is, secondpixel 22, outputs an imaging signal indicating luminance. Therefore, apixel of black and white that indicates the luminance of visible light,in other words, a monochrome image is represented by an imaging signaloutputted from a plurality of second pixels 22 included in pixel array2. This monochrome image is hereinafter referred to as a BW image. Itshould be noted that the above-described color image and the BW imageare collectively referred to as a visible light image.

Referring back to FIG. 1 again, description of depth acquisition device1 will be continued.

Processing circuit 30 computes subject information relating to a subjectby using the imaging signal outputted by solid-state imaging element 20.

Processing circuit 30 is constituted by, for example, an arithmeticprocessing unit such as a microcomputer. The microcomputer includes aprocessor (microprocessor), a memory, etc. and generates a lightemitting signal and an exposure signal by the processor executing adriving program stored in the memory. It should be noted that processingcircuit 30 may use PGA or ISP, etc. and may be constituted by onehardware or multiple hardware.

Processing circuit 30 calculates distance to a subject by, for example,a TOF distance measurement method which is performed by using theimaging signal from first pixel 21 of solid-state imaging element 20.

Hereinafter, referring to the drawings, calculation of distance to asubject by the TOF distance measurement method performed by processingcircuit 30 will be described.

FIG. 3 is a time chart illustrating a relationship between the lightemitting timing of the light emitting element of light source 10 and theexposure timing of first pixel 21 of solid-state imaging element 20 whenprocessing circuit 30 calculates a distance to a subject by using theTOF distance measurement method.

In FIG. 3, Tp is a light emission period during which a light emittingelement of light source 10 emits irradiation light, and Td is a delaytime from when the light emitting element of light source 10 emits theirradiation light until when reflected light which is the irradiationlight reflected by a subject returns to solid-state imaging element 20.And the first exposure period is at the same timing at that of the lightemission period during which light source 10 emits irradiation light,and the second exposure period is timing from the end time point of thefirst exposure period until an elapse of the light emission period Tp.

In FIG. 3, q1 indicates a total amount of exposure amount in first pixel21 of solid-state imaging element 20 by the reflected light in the firstexposure period, and q2 indicates a total amount of exposure amount infirst pixel 21 of solid-state imaging element 20 by the reflected lightin the second exposure period.

By performing light emission of irradiation light by the light emittingelement of light source 10 and exposure by first pixel 21 of solid-stateimaging element 20 at a timing shown in FIG. 3, it is possible torepresent a distance d to a subject by the following (Equation 1) with cas the speed of light.

d=c×Tp/2×q2/(q1+q2)   (Equation 1)

Therefore, processing circuit 30 can calculate the distance to a subjectby using an imaging signal from first pixel 21 of solid-state imagingelement 20 by using (Equation 1).

Further, a plurality of first pixels 21 of solid-state imaging element20 may be exposed for a third exposure period Tp after the end of thefirst exposure period and the second exposure period. The plurality offirst pixels 21 can detect noises other than reflected light by theexposure amount obtained in the third exposure period Tp. That is,processing circuit 30 can more accurately calculate the distance d to asubject by deleting noises respectively from exposure amount q1 in thefirst exposure period and exposure amount q2 in the second exposureperiod, in the above-described (Equation 1).

Referring back to FIG. 1 again, description of depth acquisition device1 will be continued.

Processing circuit 30 may perform detection of a subject, andcalculation of the distance to the subject by using imaging signalsfrom, for example, second pixel 22 of solid-state imaging element 20.

That is, processing circuit 30 may perform detection of a subject andcalculation of a distance to the subject based on visible light imageimaged by a plurality of second pixels 22 of solid-state imaging element20. Here, the detection of a subject may be implemented by, for example,performing discrimination of shape by pattern recognition through edgedetection of a singular point of the subject, or may be implemented byprocessing such as Deep Learning by using a learning model trained inadvance. Further, calculation of a distance to the subject may beperformed by using global coordinate transformation. As a matter ofcourse, detection of a subject may be implemented by multi-modallearning process by using not only visible light image, but alsoluminance and distance information of infrared light imaged by firstpixel 21.

Processing circuit 30 generates a light emission signal indicating thetiming of light emission, and an exposure signal indicating the timingof exposure. Then, processing circuit 30 outputs the generated lightemission signal to light source 10, and outputs the generated exposuresignal to solid-state imaging element 20.

Processing circuit 30 may make depth acquisition device 1 implementcontinuous imaging at a predetermined frame rate, for example, bygenerating and outputting a light emission signal so as to make lightsource 10 emit light on a predetermined cycle, and generating andoutputting an exposure signal so as to expose solid-state imagingelement 20 on a predetermined cycle. Moreover, processing circuit 30includes, for example, a processor (microprocessor), a memory, and thelike, and a light emission signal and an exposure signal are generatedby the processor executing driving program stored in the memory.

Diffusion plate 50 adjusts the intensity distribution and the angle ofirradiation light. Moreover, in the adjustment of the intensitydistribution, diffusion plate 50 makes the intensity distribution ofirradiation light from light source 10 uniform. It should be noted thatin the example shown in FIG. 1, depth acquisition device 1 includesdiffusion plate 50; however, this diffusion plate 50 may not beincluded.

Lens 60 is an optical lens that collects light entering from the outsideof depth acquisition device 1 on the surface of pixel array 2 ofsolid-state imaging element 20.

Band-pass filter 70 is an optical filter that transmits infrared lightwhich is reflected light and visible light. It should be noted that inan example shown in FIG. 1, depth acquisition device 1 includesband-pass filter 70; however, this band-pass filter 70 may not beincluded.

Depth acquisition device 1 of the above-described configuration is usedby being installed on a transport equipment. For example, depthacquisition device 1 is used by being installed on a vehicle thattravels on the road surface. It should be noted that the transportequipment on which depth acquisition device 1 is installed does not needto be limited to a vehicle. Depth acquisition device 1 may be used bybeing installed on a transport equipment other than vehicles, such asmotorcycles, boats, air planes, and the like.

[Outline of Depth Acquisition Device]

Depth acquisition device 1 in the present embodiment acquires an IRimage and a BW image with hardware configuration shown in FIG. 1 byimaging of a substantially same scene, the imaging being performed at asubstantially same viewpoint and a same time. And depth acquisitiondevice 1 corrects the depth at each position in the IR image obtainedfrom that IR image by using the BW image. Specifically, when a regionshowing dust (hereinafter, referred to as a dust region) exists in an IRimage, depth acquisition device 1 corrects the depth at each position inthe dust region obtained from that IR image by using the image in theregion of the BW image corresponding to the dust region. It is thereforepossible to appropriately acquire a depth of the dust region byexcluding influence of noise that is the dust.

FIG. 4 is a block diagram illustrating an example of a functionalstructure of depth acquisition device 1.

Depth acquisition device 1 includes light source 101, IR camera 102, BWcamera 103, depth estimator 111, and dust detector 112.

Light source 101 may be constituted by light source 10 and diffusionplate 50 shown in FIG. 1.

IR camera 102 may be constituted by a plurality of first pixels 21 ofsolid-state imaging element 20, lens 60, and band-pass filter 70 shownin FIG. 1. Such IR camera 102 acquires an IR image by performing imagingof a scene including the subject with infrared light according to timingat which light source 101 irradiates infrared light to the subject.

BW camera 103 may be constituted by a plurality of second pixels 22 ofsolid-state imaging element 20, lens 60, and band-pass filter 70 shownin FIG. 1. Such BW camera 103 acquires a visible light image(specifically, a BW image) by imaging of a substantially same scene asthat of the infrared image, the imaging being performed with visiblelight at a substantially same viewpoint and imaging time as those of theinfrared image.

Depth estimator 111 and dust detector 112 may be implemented as afunction of processing circuit 30 shown in FIG. 1, specifically as afunction of processor 110.

Dust detector 112 detects a dust region from an IR image based on an IRimage obtained by imaging by IR camera 102, and a BW image obtained byimaging by BW camera 103. In the other words, dust detector 112 dividesthe IR image obtained by the imaging into dust regions showing dust andnon-dust regions not showing dust.

When there are fine particles such as dust in the vicinity of IR camera102, the IR image shows the dust as large noise. The dust regionaccording to the present embodiment is a region having a higherluminance and showing dust. For example, when there is dust near depthacquisition device 1, infrared light which has been irradiated fromlight source 101 to dust and reflected on the dust is received bysolid-state imaging element 20 while keeping a higher luminance.Therefore, in the IR image, a region showing dust, in other words, eachpixel in the dust region has a higher luminance.

Depth estimator 111 estimates a depth at each position in the IR imageincluding a dust region detected by dust detector 112. Specifically,depth estimator 111 acquires an IR image obtained by imaging by IRcamera 102 according to irradiation timing of infrared light to asubject by light source 101, and based on the IR image, depth at eachposition in the IR image is estimated. Further, depth estimator 111corrects the depth at each position estimated in the dust regiondetected by dust detector 112, based on the BW image. That is, depthestimator 111 estimates a depth of the dust region based on the IRimage, the BW image, and the dust region.

FIG. 5 is a block diagram illustrating another example of a functionalstructure of depth acquisition device 1.

Depth acquisition device 1 may include memory 200 and processor 110.

Moreover, processor 110 may not only include depth estimator 111 anddust detector 112, may but also include light emission timing acquirer113, IR image acquirer 114, and BW image acquirer 115. It should benoted that these components are implemented respectively as a functionof processor 110.

Light emission timing acquirer 113 acquires timing informationindicating a timing at which light source 101 irradiates infrared lightto a subject. That is, light emission timing acquirer 113 outputs thelight emission signal shown in FIG. 1 to light source 101, and therebyacquires information indicating the timing of the output as theabove-described timing information. IR image acquirer 114 acquires an IRimage which is retained in memory 200, the IR image being obtained byimaging of a scene including a subject with infrared light according tothe timing indicated by the timing information.

BW image acquirer 115 acquires a BW image retained in memory 200, inwhich the BW image is obtained by imaging of a substantially same sceneas that of the above-described IR image with visible light, the imagingbeing performed at a substantially same viewpoint and imaging time asthose of the IR image.

Dust detector 112 detects, as described above, a dust region from an IRimage, and depth estimator 111 estimates a depth based on the IR image,the BW image, and the dust region.

It should be noted that depth acquisition device 1 in the presentEmbodiment may be constituted by processor 110 and memory 200 withoutincluding light source 101, IR camera 102, and BW camera 103.

FIG. 6 is a flowchart illustrating overall processing operation of depthacquisition device 1.

(Step S11)

First, light source 101 emits light, and thereby irradiates infraredlight to a subject.

(Step S12)

Next, IR camera 102 acquires an IR image. That is, IR camera 102 imagesa scene including a subject which is irradiated with infrared light bylight source 101. In this way, IR camera 102 acquires an IR image basedon infrared light reflected from the subject. Specifically, IR camera102 acquires IR images obtained at respective timings and by exposureamounts of the first exposure period, the second exposure period, andthe third exposure period shown in FIG. 3.

(Step S13)

Next, BW camera 103 acquires a BW image. That is, BW camera 103 acquiresa BW image corresponding to the IR image acquired in step S12, that is,a BW image of the same scene, the same viewpoint, and the same imagingtime as those of the IR image.

(Step S14)

Then, dust detector 112 detects a dust region from the IR image acquiredin step S12.

(Step S15)

Next, depth estimator 111 estimates a depth of the dust region based onthe IR image acquired in step S12, the BW image acquired in step S13,and the dust region detected in step S14.

FIG. 7 is a flowchart illustrating overall processing operation byprocessor 110 of depth acquisition device 1.

(Step S21)

First, light emission timing acquirer 113 of processor 110 acquirestiming information indicating the timing at which light source 101irradiates infrared light to a subject.

(Step S22)

Next, IR image acquirer 114 acquires an IR image from IR camera 102 thathas performed imaging according to the timing indicated by the timinginformation acquired in step S21. For example, IR image acquirer 114outputs an exposure signal to IR camera 102 at the timing at which thelight emission signal shown in FIG. 1 is outputted from light emissiontiming acquirer 113. In this way, IR image acquirer 114 causes IR camera102 to start imaging, and acquires the IR image obtained by the imagingfrom IR camera 102. At this moment, IR image acquirer 114 may acquire anIR image from IR camera 102 via memory 200, or directly from IR camera102.

(Step S23)

Next, BW image acquirer 115 acquires a BW image corresponding to the IRimage acquired in step S22 from BW camera 103. At this moment, BW imageacquirer 115 may acquire the BW image from BW camera 103 via memory 200,or directly from BW camera 103.

(Step S24)

Then, dust detector 112 detects a dust region from the IR image.

(Step S25)

Next, depth estimator 111 estimates a depth of a dust region based onthe IR image acquired in step S22, the BW image acquired in step S23,and the dust region detected in step S24. As a result of this, depthinformation which at least indicates a depth of the dust region iscalculated. It should be noted that at this moment, depth estimator 111may estimate depth of not only the dust region but also the entire IRimage, and calculate depth information indicating the estimation result.

Specifically, depth estimator 111 in the present embodiment estimates,from the IR image acquired in step S22, a depth at each position in theIR image. Then, depth estimator 111 corrects the depth at each positionin the dust region by using the BW image. It should be noted that eachposition may be respective positions of a plurality of pixels, or aposition of a block consisting of a plurality of pixels.

In such depth acquisition device 1 in the present embodiment, since adust region is detected from an IR image, and in that dust region, depthis estimated based on not only the IR image but also the BW image, it ispossible to appropriately acquire the depth of the dust region. That is,the scene to be imaged is substantially the same between the IR imageand the BW image, and the viewpoint and the imaging time are alsosubstantially the same. Therefore, the IR image and the BW image havehigh correlation. Moreover, dust and the like are a phenomenon which isdependent on wavelength, and even if dust and the like occur in an IRimage, it is highly likely that dust does not occur in the BW image evenif it occurs in the IR image. Therefore, it is possible to supplementinformation lacking in a dust region from a region (that is,corresponding region) in the BW image corresponding the dust region.Consequently, it is possible to appropriately acquire the depth of thedust region.

[Specific Functional Structure of Depth Acquisition Device]

FIG. 8 is a block diagram illustrating a specific functional structureof processor 110 of depth acquisition device 1.

Processor 110 includes first depth estimator 111 a, second depthestimator 111 b, dust detector 112, higher-luminance-region detector116, FOE detector 117, first edge detector 117IR, second edge detector117BW, and outputter 118. It should be noted that first depth estimator111 a and second depth estimator 111 b correspond to depth estimator 111shown in FIG. 5. Moreover, processor 110 may include the above-describedlight emission timing acquirer 113, IR image acquirer 114, and BW imageacquirer 115.

Higher-luminance-region detector 116 detects a region having a luminancenot less than first threshold in an IR image, as a higher luminanceregion.

FOE detector 117 detects a focus of expansion (FOE) in the BW image. FOEis also referred to as a vanishing point. It is known that when IRcamera 102 moves in parallel directions while the subject is still, anoptical flow that is virtual movement on the image intersects one point.The one point is an FOE.

Dust detector 112 determines, for each of at least one higher luminanceregion in the IR image, whether or not the higher luminance region is adust region. Here, the inventors have found that each dust shown in theIR image is seen long along a straight line or an arc which intersectsan FOE of the higher luminance region, as if the dust is blown out fromthe FOE, or the dust is located along the straight line or the arc. WhenIR camera 120 is set in a movable body such as an automobile, movementof dust is smaller than movement of IR camera 120, which allows toassume that the dust is still. Therefore, IR image shows the dust thatlooks blown out from the FOE.

Furthermore, in depth acquisition device 1 according to the presentembodiment, in terms of noise cancellation, each frame in the IR imageand the BW image is obtained by imaging in which light exposure andlight shielding are repeated a plurality of times. Therefore, by theexposure processes at different timings in a frame cycle, the same dustappears at a plurality of positions on one frame. Since the dust on theimage is moving as if the dust is blown out from the FOE, dust regionsat respective positions and the FOE are located on a straight line.Furthermore, when IR camera 120 imaging the same dust moves slowly, aplurality of dust regions resulted from a plurality of exposureprocesses in one frame cycle overlap in the IR image. As a result, onedust region having a shape that looks like leaving trace is formed. Inthis case, the FOE exists in the direction of leaving the trace as seenfrom the dust region. Dust detector 112 detects dust regions by usingthe above-described characteristics.

In other words, when a higher luminance region having a luminance notless than the first threshold in the IR image satisfies firstconditions, dust detector 112 according to the present applicationdetects the higher luminance region as a dust region. More specifically,the first conditions are that the center of gravity of the higherluminance region is located on a straight line or an arc whichintersects (i) the center of gravity of each of at least two higherluminance regions that are different from the higher luminance region inthe IR image and (ii) a focus of expansion (FOE) of the IR image or theBW image. It is also possible that the first conditions are that aprinciple axis or a line extending from the principle axis of the higherluminance region intersects the FOE of the IR image or the BW image. Itshould be noted that the principle axis is an axis of the trace when thehigher luminance region has a shape of leaving the trace. It istherefore possible to detect dust regions at high accuracy.

Furthermore, dust detector 112 may detect a dust region by using alsothe characteristics that a dust region is not observed in the BW imageas if the dust region is observed in the IR image. In other words, it ispossible that when a higher luminance region further satisfies thesecond conditions, dust detector 112 detects the higher luminance regionas a dust region. For example, the second conditions are that aluminance at a position in the BW image which corresponds to the centerof gravity of the higher luminance region in the IR image is less than asecond threshold. It is also possible that the second conditions arethat a correlation coefficient between a luminance in the higherluminance region in the IR image and a luminance in a region in the BWimage which corresponds to the higher luminance region in the IR imageis less than a third threshold. It should be noted that a region in theBW image which corresponds to the higher luminance region in the IRimage is located spatially at the same position as the position of thehigher luminance region in the IR image, and has the same shape and sizeas those of the higher luminance region in the IR image. It is thereforepossible to detect dust regions at high accuracy.

First depth estimator 111 a and second depth estimator 111 b have afunction as the above-described depth estimator 111.

First depth estimator 111 a estimates, based on an IR image acquiredaccording to the timing of irradiation of infrared light by light source101, a depth at each position in the IR image. First depth estimator 111a outputs the information indicating the estimated depth at eachposition in the IR image, as the first depth information. That is, firstdepth estimator 111 a estimates first depth information which indicatesthe depth at each position in the IR image.

Second depth estimator 111 b corrects the first depth information basedon the BW image and the dust region in the IR image. As a result ofthis, in the depth at each position in the IR image indicated by thefirst depth information, the depth of the dust region is corrected.Second depth estimator 111 b outputs information indicating a correcteddepth at each position in the dust region as the second depthinformation. That is, second depth estimator 111 b estimates the seconddepth information indicating corrected depth at each position in thedust region by correcting depth at each position in the dust regionindicated by the first depth information based on the BW image.

Outputter 118 replaces a depth at each position in the dust regionindicated by the first depth information with a corrected depth at eachposition in the dust region indicated by the second depth information.As a result of this, third depth information is generated, whichincludes a depth at each position in the region other than the dustregion of the IR image indicated by the first depth information, and acorrected depth at each position in the dust region of the IR imageindicated by the second depth information. Outputter 118 outputs thethird depth information.

As a result of this, the third depth information indicates a depthobtained from an IR image as the depth outside the dust region of the IRimage, and indicates a depth obtained from the IR image and correctedbased on the BW image as the depth of the dust region of the IR image.Therefore, in the present embodiment, even when there is a dust regionin the IR image, it is possible to appropriately estimate the depth ofthe entire IR image.

FIG. 9A shows an example of IR image. FIG. 9B shows an example of BWimage.

As shown in FIG. 9B, in the BW image, for example, a scene in which thesurroundings of the road on which a vehicle runs are moved away from BWcamera 103 set in the vehicle is imaged by BW camera 103. On the road,for example, the running of the vehicle whirls up the dust. Accordingly,when IR camera 102 images the same scene as the scene shown in FIG. 9Bfrom the same viewpoint and at the same time as those of BW camera 103,the IR image shown in FIG. 9A is acquired.

In the IR image acquired as described above, as shown in FIG. 9A,regions having a higher luminance have occurred. These regions includeregions showing dust, namely, a dust region. For example, the dustregions are seen in a center part and a right part of the IR image. Onthe other hand, the BW image does not show such dust. This is because,in imaging of the IR image, the infrared light from light source 101 isirregularly reflected on the dust in the vicinity of IR camera 102 andBW camera 103, while in imaging of the BW image, the influence of theirregular reflection is smaller. Therefore, dust detector 112 detectsdust regions by using characteristics that dust regions are observed inthe IR image while no dust image is observed in the BW image.

FIG. 10 shows an example of binarized image obtained by binarization ofan IR image.

Higher-luminance-region detector 116 detects, in the IR image shown inFIG. 9A, a region having a luminance of not less than a first thresholdas a higher luminance region. That is, higher-luminance-region detector116 binarizes the luminance at each position (that is, at each pixel) inthe IR image. Consequently, for example, as shown in FIG. 10, abinarized image consisting of white regions and black regions (hatchedregion in FIG. 10) is generated.

FIG. 11 shows an example of a candidate of dust region in an IR image.

Dust detector 112 first detects the center of gravity of a higherluminance region, which is a white region, in a binarized image. Suchdetection of the center of gravity is performed for each of all thehigher luminance regions in the binarized image. Next, for each of allthe higher luminance regions in the binarized image, dust detector 112discriminates whether or not the luminance of the position in the BWimage corresponding to the center of gravity of the higher luminanceregion (hereinafter referred to as a center of gravity correspondingposition) is less than a second threshold. That is, dust detector 112discriminates whether or not the higher luminance region satisfies theabove-described second condition. Consequently, dust detector 112decides, as a candidate for the dust region, a higher luminance regioncorresponding to the center of gravity corresponding position which hasbeen discriminated to have a luminance of less than the secondthreshold. For example, as shown in FIG. 11, dust detector 112 detectseach of the five higher luminance regions A to E as candidates for thedust region. That is, the IR image or the binarized image is regionallysegmented into five dust candidate regions A to E and a non-dust regionthat is not a dust region.

FIG. 12 shows an example of FOE which is detected for a BW image.

FOE detector 117 detects optical flows from, for example, amultiple-frame BW image including, for example, the BW image shown inFIG. 9B, and detects an FOE by finding the intersection of the opticalflows by robust estimation such as Random Sample Consensus (RANSAC).

FIG. 13 shows an example of a principal axis to be detected for dustcandidate regions A to E.

Dust detector 112 detects the principal axis for each of the dustcandidate regions A to E detected as shown in FIG. 11. Specifically,dust detector 112 detects the first principal component axis as theprincipal axis of the dust candidate region by performing principalcomponent analysis on each pixel of the dust candidate region in the IRimage or the binarized image.

FIG. 14 shows an example of a dust region and a non-dust region.

For each of the dust candidate regions A to E in the IR image or thebinarized image, dust detector 112 determines whether or not theprincipal axis of the dust candidate region or an extension line of theprincipal axis intersects the FOE detected by FOE detector 117. That is,dust detector 112 determines whether or not the dust candidate regionsatisfies the above-described first condition. Then, dust detector 112detects, as a dust region, a dust candidate region having a principalaxis or an extension line that intersects the FOE, and detects, as anon-dust region, a dust candidate region having a principal axis or anextension line that does not intersect the FOE. For example, as shown inFIG. 14, the extension line of the principal axis of each of dustcandidate regions B, C, and E intersect the FOE, and the principal axisand its extension line of each of dust candidate regions A and D do notintersect the FOE. Therefore, dust detector 112 detects dust candidateregions B, C, and E as dust regions, and detects dust candidate regionsA and D as non-dust regions.

FIG. 15A shows another example of IR image. FIG. 15B shows anotherexample of BW image.

The BW image shown in FIG. 15B is an image obtained by imaging at atiming different from that of the BW image shown in FIG. 9B. In the BWimage shown in FIG. 15B, as in the example shown in FIG. 9B, a scene isprojected in which the surroundings of the road on which the vehicle istraveling approaches BW camera 103, by the imaging with BW camera 103attached to the vehicle. In addition, dust is flying up on this road,for example, due to travelling of a vehicle. Therefore, when IR camera102 images the same scene as the scene shown in FIG. 15B at the sameviewpoint and the same time as those of BW camera 103, the IR imageshown in FIG. 15A is acquired.

As shown in FIG. 15A, the IR image acquired in this way has regions ofhigher luminance. These regions include regions where dust is projected,that is, dust regions. For example, it is confirmed that the dust regionexists in the left part of the IR image. On the other hand, as in theexample shown in FIG. 9B, no dust is projected on the BW image.

FIG. 16 shows an example of a binarized image obtained by binarizing anIR image.

In the IR image shown in FIG. 15A, higher-luminance-region detector 116detects a region having a luminance not less than the first threshold,as a higher luminance region. That is, higher-luminance-region detector116 binarizes the luminance at each position (that is, each pixel) inthe IR image. Consequently, for example, as shown in FIG. 16, abinarized image consisting of a white region and a black region (hatchedregion in FIG. 16) is generated.

FIG. 17 shows an example of a candidate of dust region in an IR image.

Dust detector 112 first detects the center of gravity of a higherluminance region, which is a white region in the binarized image. Suchdetection of the center of gravity is performed for each of all thehigher luminance regions in the binarized image. Next, dust detector 112discriminates, for each of all the higher luminance regions in thebinarized image, whether or not the luminance of the position in the BWimage corresponding to the center of gravity of the higher luminanceregion (that is, center of gravity corresponding position) is less thanthe second threshold. That is, dust detector 112 discriminates whetheror not the higher luminance region satisfies the above-described secondcondition. Consequently, dust detector 112 decides, as a candidate forthe dust region, a higher luminance region corresponding to the centerof gravity corresponding position which has been discriminated to have aluminance of less than the second threshold. For example, as shown inFIG. 17, dust detector 112 detects, as a candidate for the dust region,each of region group A consisting of a plurality of higher luminanceregions, higher luminance region B, region group C consisting of aplurality of higher luminance regions, and higher luminance region D.That is, the IR image or binarized image is regionally segmented intodust candidate regions and non-dust regions each of which is not a dustregion. The dust candidate regions consist of each region included inregion group A, region B, each region included in region group C, andregion D.

FIG. 18 shows an example of FOE which is detected for a BW image.

FOE detector 117 detects optical flows from, for example, amultiple-frame BW image including, for example, the BW image shown inFIG. 15B, and detects an FOE by finding the intersection of the opticalflows by robust estimation such as Random Sample Consensus (RANSAC).

FIG. 19 shows an example of arrangement of each dust candidate region.

For example, when the speed of a vehicle on which IR camera 102 ismounted is high, the same dust is projected on a plurality of places inone frame of IR image due to exposures at mutually different timings inthe frame cycle, as described above. Since the dust is moving in such away to be blown out from the FOE, the dust regions which are located ineach of the plurality of places, and the FOE are arranged on a straightline. Further, when a wide-angle lens or a fisheye lens is used as thelens of IR camera 102, the distortion of the lens is large. When theinfluence of lens distortion is large, the optical flows, which areapparent movement on the screen, do not intersect at one point, so thateach dust region and FOE do not exist on a straight line. In this way,when the distortion of the lens is large, each dust region and FOE arearranged on an arc due to the influence of lens distortion.

In the case of the example shown in FIG. 19, dust detector 112determines whether or not each of each dust candidate region of regiongroup A, dust candidate region B, each dust candidate region of regiongroup C, and dust candidate region D, which are detected as shown inFIG. 17, are arranged on arc along with the FOE. That is, when makingdetermination for one dust candidate region, dust detector 112determines whether or not the center of gravity of the dust candidateregion to be determined is arranged on an arc which intersects thecenter of gravity of each of at least two other dust candidate regionsdifferent from the dust candidate region to be determined, and the FOEof the BW image. That is, dust detector 112 determines whether or not ahigher luminance region, which is a dust candidate region, satisfies theabove-described first condition. If the center of gravity of the dustcandidate region to be determined is arranged on the arc, dust detector112 detects the dust candidate region as the dust region. On thecontrary, if the center of gravity of the dust candidate region to bedetermined is not arranged on the arc, dust detector 112 detects thedust candidate region as the non-dust region.

Therefore, in the example shown in FIG. 19, since each of the pluralityof dust candidate regions included in region group A and the FOE arearranged on an arc, dust detector 112 detects, as the dust region, eachof the plurality of dust candidate regions included in region group A.Similarly, since each of the plurality of dust candidate regionsincluded in region group B and the FOE are arranged on an arc, dustdetector 112 also detects, as the dust region, each of the plurality ofdust candidate regions included in region group B. On the other hand,since each of dust candidate regions B and D is not arranged on the arcintersecting at least two other dust candidate regions and the FOE, dustdetector 112 detects, as the non-dust region, each of dust candidateregions B and D.

It is noted that when the distortion of lens is large, dust detector 112may perform distortion correction processing for the BW image and the IRimage, which have been imaged. For example, dust detector 112 mayperform distortion correction processing by using a camera calibrationmethod such as Non Patent Literature (R. Tsai, “A versatile cameracalibration technique for high-accuracy 3D machine vision metrologyusing off-the-shelf TV cameras and lenses”, IEEE Journal on Robotics andAutomation, Vol. 3, Iss. 4, pp. 323-344, 1987). In this case, if thedust candidate region to be determined is arranged on a straight lineintersecting at least two other dust candidate regions and the FOE in anIR image or its binarized image which has been subjected to thedistortion correction processing, dust detector 112 detects, as the dustregion, the dust candidate region to be determined.

FIG. 20 shows a simulation result of depth acquisition device 1.

Depth acquisition device 1 acquires a BW image shown in (a) of FIG. 20by imaging with BW camera 103, and further acquires an IR image shown in(b) of FIG. 20 by imaging with IR camera 102. These BW and IR images areimages obtained by imaging the same scene at the same viewpoint and atthe same time. In the example shown in (b) of FIG. 20, some dust regionsexist in the IR image.

First depth estimator 111 a generates first depth information shown in(c) of FIG. 20 by estimating depth from the IR image. This first depthinformation is expressed as a first depth image which indicates thedepth at each position in the IR image by luminance. In this first depthimage, the depth of the dust region is inappropriately expressed.

Second depth estimator 111 b corrects the inappropriate depth in thedust region. Then, as shown in (e) of FIG. 20, outputter 118 generatesthird depth information which indicates the corrected depth of the dustregion and the depth of the non-dust region. Like the first depthinformation, this third depth information is also expressed as a thirddepth image in which the depth is indicated by luminance. It is notedthat second depth estimator 111 b may also correct the depth of thenon-dust region in the first depth image based on the correspondingregion of the BW image.

In this way, in depth acquisition device 1 in the present embodiment, itis possible to bring the third depth image closer to the correct depthimage shown in (d) of FIG. 20 in the entire image including dustregions.

[Specific Processing Flow of Depth Acquisition Device]

FIG. 21 is a flowchart illustrating overall processing operation ofdepth acquisition device 1 shown in FIG. 8.

(Step S31)

First, higher-luminance-region detector 116 detects a higher luminanceregion from an IR image.

(Step S32)

Dust detector 112 decides whether or not the higher luminance region isa dust candidate region. As a result of this, the IR image is dividedinto dust candidate regions and non-dust regions.

(Step S33)

FOE detector 117 detects FOE by using a BW image.

(Step S34)

Dust detector 112 detects a dust region based on the dust candidateregions and FOE. As a result of this, the IR image is divided into dustregions and non-dust regions.

(Step S35)

First depth estimator 111 a generates first depth information from theIR image by using, for example, TOF.

(Step S36)

Second depth estimator 111 b generates second depth informationindicating the depth of dust regions, based on the first depthinformation of the IR image and the BW image.

(Step S37)

Outputter 118 generates third depth information by replacing the depthof dust regions indicated by the first depth information with the depthindicated by the second depth information.

FIG. 22 is a flowchart illustrating an example of detailed processing ofsteps S31 to S34 of FIG. 21.

(Step S41)

First, higher-luminance-region detector 116 determines whether or notthe luminance at each position in the IR image is not less than a firstthreshold. Here, the first threshold may be about 256, for example, ifthe IR image is an image with 12-bit gradation. Of course, this firstthreshold may be a value that varies according to environmentalconditions or settings of IR camera 102. For example, when a dark scenesuch as night is imaged, since the luminance of the entire IR image willbe lower, the first threshold may be a smaller value than when a brightscene in daytime is imaged. Moreover, when the exposure time of IRcamera 102 is long, since the luminance of the entire IR image will behigher, the first threshold may be a larger value than when the exposuretime is short.

(Step S42)

Here, upon determining that the luminance at any position is not equalto or greater than the first threshold (No in step S41),higher-luminance-region detector 116 determines that dust is notprojected on the IR image. That is, the entire IR image is determined asa non-dust region.

(Step S43)

On the other hand, upon determining that the luminance at any positionis not less than the first threshold (Yes in step S41),higher-luminance-region detector 116 performs regional segmentation ofthe IR image. That is, higher-luminance-region detector 116 segments theIR image into at least one higher luminance region and a region otherthan the higher luminance region. For this regional segmentation, amethod based on luminance such as Super Pixel may be used. It is notedthat, higher-luminance-region detector 116 may perform a filteringprocess taking advantage of the size of the region for that regionalsegmentation. For example, if the number of pixels in a higher luminanceregion is not more than a predetermined number, higher-luminance-regiondetector 116 may delete the higher luminance region. That is, even ifhigher-luminance-region detector 116 detects a higher luminance region,when the number of pixels in the region is small, the higher luminanceregion may be reclassified into a region other than the higher luminanceregion.

(Step S44)

Next, for each of the at least one higher luminance region detected bythe regional segmentation in step S43, dust detector 112 detects thecenter of gravity of the higher luminance region. Specifically, dustdetector 112 detects the center of gravity of a higher luminance regionby calculating an average value of each of the X-axis coordinatepositions and the Y-axis coordinate positions of a plurality of pixelsincluded in the higher luminance region.

(Step 545 a)

Dust detector 112 determines whether or not the luminance of theposition in the BW image corresponding to the center of gravity of thehigher luminance region (that is, a center of gravity correspondingposition) is less than the second threshold. That is, dust detector 112determines whether or not the higher luminance region satisfies thesecond condition. Upon determining that the luminance is not less thanthe second threshold (No in step 545 a), dust detector 112 discriminatesthe higher luminance region as the non-dust region (step S42). That is,in this case, it is estimated that a subject having a high lightreflectance is projected on each of the higher luminance region of theIR image and the region corresponding to the higher luminance region inthe BW image. Therefore, in this case, the higher luminance region isdiscriminated as a non-dust region. It is noted that the secondthreshold may be, for example, about 20,000 if the BW image is an imagewith 12-bit gradation. Of course, this second threshold may be a valuethat varies according to the environmental conditions or settings of BWcamera 103. For example, when a dark scene such as night is imaged,since the luminance of the entire BW image will be lower, the secondthreshold may be a smaller value than when a bright scene in daytime isimaged. Moreover, when the exposure time of BW camera 103 is long, sincethe luminance of the entire BW image will be higher, the secondthreshold may be a larger value than when the exposure time is short.

(Step S46)

On the other hand, when dust detector 112 determines that the luminanceof the center of gravity corresponding position is less than the secondthreshold (Yes in step 545 a), FOE detector 117 detects an FOE based onthe BW image.

(Step S47 a)

Dust detector 112 determines whether or not the centers of gravity ofthree or more dust candidate regions detected in step S44 and the FOEare arranged on a straight line. That is, dust detector 112 determineswhether or not the dust candidate region satisfies the above-describedfirst condition. Specifically, dust detector 112 performs fitting of thecenter of gravity of each of the three or more dust candidate regionswith a straight line intersecting the FOE, and determines whether or notan error (that is, a distance) between the straight line and each centerof gravity is not more than a permissible value. Thereby, it isdetermined whether or not the center of gravity of each of the three ormore dust candidate regions and the FOE are arranged on a straight line.If the error is not more than the permissible value, it is determinedthat the centers of gravity of the dust candidate regions and the FOEare arranged on a straight line, and if the error is not equal to orless than the permissible value, it is determined that the centers ofgravity of the dust candidate regions and the FOE are not arranged on astraight line.

(Step S50)

Upon determining that the center of gravity of each dust candidateregion and FOE are arranged on a straight line (Yes in step 547 a), dustdetector 112 discriminates, as the dust region, those dust candidateregions.

(Step S48)

On the other hand, upon determining that the centers of gravity of threeor more dust candidate regions and FOE are not arranged on a straightline (No in step S47), dust detector 112 detects a principal axis ofeach dust candidate region.

(Step S49)

Next, dust detector 112 determines whether or not the principal axis orits extension line of each dust candidate region detected in step S48intersects the FOE. That is, dust detector 112 determines whether or notthe dust candidate region satisfies another first condition differentfrom the first condition in step S47 a. Here, upon determining that theprincipal axis or its extension line intersects the FOE (Yes in stepS49), dust detector 112 discriminates, as the dust region, the dustcandidate region having the principal axis (step S50). On the otherhand, upon determining that the principal axis or its extension linedoes not intersect the FOE (No in step S49), dust detector 112discriminates, as the non-dust region, the dust candidate region havingthe principal axis (step S42).

In such a method, an IR image and a BW image whose viewpoint positionsare substantially the same are required. In depth acquisition device 1of the present embodiment, the filter to be used for each pixel is setto either an IR filter or a BW filter for each pixel. That is, as shownin FIG. 2, first pixel 21 having an IR filter and second pixel 22 havinga BW filter are alternately arranged in the column direction. As aresult of this, the IR image and the BW image at substantially the sameviewpoint and the same time can be acquired so that it is possible toappropriately discriminate the dust region.

FIG. 23 is a flowchart illustrating another example of detailedprocessing of steps S31 to S34 of FIG. 21. The flowchart shown in FIG.23 includes step S47 b in place of step S47 a of each step of theflowchart of FIG. 22.

(Step S47 b)

For example, when the distortion of the lens of IR camera 102 is large,each dust region and the FOE are not arranged on a straight line, butare arranged on an arc according to the lens distortion as describedabove.

Therefore, dust detector 112 may determine whether or not the centers ofgravity of the three or more dust candidate regions detected in step S44and the FOE are arranged on an arc. Specifically, dust detector 112obtains an approximation curve of respective centers of gravity of thethree or more dust candidate regions and the FOE, and determines whetheror not the error (that is, the distance) between the approximation curveand each center of gravity is not more than a permissible value. As aresult, it is determined whether or not the center of gravity of each ofthe three or more dust candidate regions and the FOE are arranged on anarc. That is, if the error is not more than the permissible value, it isdetermined that the centers of gravity of those dust candidate regionsand the FOE are arranged on an arc, and if the error is not equal to orless than the permissible value, it is determined that the centers ofgravity of those dust candidate regions and the FOE are not arranged onthe arc. The above-described approximation curve is represented by acurve of an order of the number which is not more than the number ofthree or more dust candidate regions to be determined.

Further, in step S46, FOE detector 117 may detect optical flows from aplurality of IR images instead of the BW image, obtain an intersectionof the optical flows by robust estimation such as RANSAC, and detect theintersection as FOE. Further, FOE detector 117 may detect the movementof IR camera 102 or BW camera 103, and detect FOE through calculationusing the movement and internal parameters of IR camera 102 or BW camera103.

FIG. 24 is a flowchart showing an example of alternative processing ofsteps S31 to S34 in FIG. 21. In the flowchart shown in FIG. 24, step S32of FIG. 21 is omitted. In other words, in the flowchart shown in FIG.24, step S45 a of the flowchart of FIG. 22 is omitted.

That is, in the examples shown in FIGS. 21 to 23, dust detector 112determines whether or not a higher luminance region is the dustcandidate region by taking advantage of the property that dust isobserved in the IR image but not in the BW image. That is, as shown instep S45 a of each of FIGS. 22 and 23, dust detector 112 determineswhether or not the higher luminance region is the dust candidate regionbased on the luminance of the center of gravity corresponding positionin the BW image. However, as shown in the flowchart shown in FIG. 24,dust detector 112 does not have to perform the determination in step S45a. In this case, any higher luminance region obtained by the regionalsegmentation in step S43 is treated as a dust candidate region.

FIG. 25 is a flowchart showing another example of the detailedprocessing of steps S31 to S34 of FIG. 21. The flowchart shown in FIG.25 includes step S45 binstead of step S45 a in each step of theflowchart of FIG. 22.

In order to determine the dust candidate region, the property that dustis observed in an IR image but not in a BW image may be used morepositively. For example, the correlation coefficient between the IRimage and the BW image may be used as the property. As described above,if dust exists, the dust is projected on the IR image, but the dust isnot projected on the BW image. That is, since a distant subject isprojected on the BW image, the images are significantly differentbetween the IR image and the BW image each of which has the sameviewpoint position. Therefore, by judging whether or not the projectedsubjects are equal by using a correlation coefficient between the IRimage and the BW image, it is possible to discriminate whether or noteach higher luminance region is the dust candidate region.

(Step 545 b)

In step S45 b, dust detector 112 calculates a correlation coefficient ofluminance between the higher luminance region of the IR image and theregion in the BW image corresponding to the higher luminance region(that is, the corresponding region) for each of at least one higherluminance region obtained by the regional segmentation in step S43. Thecorrelation coefficient is obtained by arranging the luminance of eachpixel of the IR image and the BW image in a vector shape for eachregion, calculating the inner product value thereof, and normalizing itwith the number of pixels. That is, dust detector 112 normalizes theinner product value between the vector consisting of the luminance ofeach pixel in the higher luminance region of the IR image and the vectorconsisting of the luminance of each pixel in the corresponding region ofthe BW image. As a result, the correlation coefficient for the higherluminance region is calculated.

Then, dust detector 112 determines whether or not the calculatedcorrelation coefficient is less than the third threshold. That is, dustdetector 112 determines whether or not the higher luminance regionsatisfies the above-described second condition. Here, if the correlationcoefficient is not less than the third threshold (No in step 545 b), itis highly likely that the correlation coefficient is high because thesame subject is projected on the higher luminance region of the IR imageand the corresponding region of the BW image. Therefore, in this case,dust detector 112 discriminates that the higher luminance region is notthe dust region (step S42). On the other hand, when the correlationcoefficient is less than the third threshold (Yes in step 545 b), thereis possibility that the correlation coefficient is low because differentsubjects are projected on the higher luminance region of the IR imageand the corresponding region of the BW image. Therefore, in this case,dust detector 112 discriminates that the higher luminance region is thedust candidate region (step S50).

Further, learning process may be used to implement the discriminationbetween the dust region and the non-dust region. For the learningprocess, for example, a process such as Deep Learning may be used. Inthis case, for performing learning, an IR image and a BW image, and acorrect image in which the IR image is segmented into dust regions andnon-dust regions are prepared in advance. Next, the IR image and the BWimage are given to the learning model as inputs. Then, the learningmodel is trained such that the output from the learning model withrespect to the input matches the correct image. The learning model is,for example, a neural network. The output from the learning model is animage whose each pixel indicates a numeral “0” or a numeral “1”, wherethe numeral “0” indicates that that pixel belongs to a non-dust region,and the numeral “1” indicates that that pixel belongs to a dust region.

By using the learning model that has been trained in advance in thisway, dust detector 112 discriminates between a dust region and anon-dust region. That is, dust detector 112 inputs an IR image and a BWimage to the learning model as inputs. Then, dust detector 112discriminates the region including the pixel corresponding to thenumeral “0” outputted from the5 learning model, as the non-dust region.Further, dust detector 112 discriminates the region including the pixelcorresponding to the numeral “1” outputted from the learning model, asthe dust region.

By the processing described so far, dust detector 112 segments an imagedIR image into a dust region in which the dust is projected and anon-dust region in which the dust is not projected.

<Depth Correction Processing>

Second depth estimator 111 b generates second depth information by usinga BW image, first depth information, and a dust region (that is,discrimination result of the above-described region).

As described above, the noise caused by dust is a phenomenon caused bydiffused reflection of infrared light irradiated from light source 101.Therefore, dust that is projected as noise on the IR image is often notprojected on the BW image. Accordingly, it is possible to acquire seconddepth information which is free from effects of the dust projected onthe IR image by correcting the first depth information, only for dustregions, by using the BW image instead of the first depth informationobtained from the IR image.

For the acquisition of the second depth information, a guided filterwhich is a type of image correction filter may be used. The guidedfilter is disclosed in Non Patent Literature (Kaiming He, Jian Sun andXiaoou Tang, “Guided Image Filtering”, IEEE Transactions on PatternAnalysis and Machine Intelligence, Vol. 35, Iss. 6, pp. 1397-1409,2013). The guided filter is a filter that corrects a target image byusing correlation between the target image and a reference image. In theguided filter, it is supposed that a reference image I and a targetimage p are represented by parameters a and b as indicated by thefollowing (Equation 2).

q _(i) =a _(k) I _(i) +b _(k) , ∀i ∈ ω _(k)   (Equation 2)

Where, q indicates an output image obtained by correcting target imagep, i indicates the number of each pixel, and wk indicates a peripheralregion of pixel k. Moreover, parameters a and b are represented by thefollowing (Equation 3).

$\begin{matrix}{{a_{k} = \frac{{\frac{1}{\omega }{\sum_{i \in \omega_{k}}{I_{i}p_{i}}}} - {\mu_{k}{\overset{\_}{p}}_{k}}}{\sigma_{k}^{2} + \epsilon}}{b_{k} = {{\overset{\_}{p}}_{k} - {a_{k}\mu_{k}}}}} & \left( {{Equation}\mspace{14mu} 3} \right)\end{matrix}$

Where, ϵ is a regularization parameter. Further, μ and σ are the meanand variance in a block of reference image, and are calculated by thefollowing (Equation 4).

$\begin{matrix}{{\mu_{k} = {\frac{1}{\omega }{\sum_{i \in \omega_{k}}I_{i}}}}{\sigma_{k}^{2} = {\frac{1}{\omega }{\sum_{i \in \omega_{k}}\left( {I_{i} - \mu_{k}} \right)^{2}}}}} & \left( {{Equation}\mspace{14mu} 4} \right)\end{matrix}$

Where, to suppress noises included in the obtained parameters a and b,the output is obtained as shown in the following (Equation 5) by usingaveraged parameters.

$\begin{matrix}{{q_{i} = {{{\overset{\_}{a}}_{k}I_{i}} + {\overset{\_}{b}}_{k}}}{{\overset{\_}{a}}_{k} = {\frac{1}{\omega }{\sum_{k \in \omega_{i}}a_{k}}}}{{\overset{\_}{b}}_{k} = {\frac{1}{\omega }{\sum_{k \in \omega_{i}}b_{k}}}}} & \left( {{Equation}\mspace{14mu} 5} \right)\end{matrix}$

In the present embodiment, second depth estimator 111 b corrects thefirst depth information (or the first depth image) which is the targetimage by giving a BW image as the reference image. As a result of this,the second depth information is generated or acquired. In order togenerate such second depth information, an IR image and a BW imagehaving substantially the same viewpoint position are required. In depthacquisition device 1 of the present embodiment, for each pixel, thefilter used for the pixel is set to either the IR filter or the BWfilter. That is, as shown in FIG. 2, first pixel 21 having an IR filterand second pixel 22 having a BW filter are alternately arranged in thecolumn direction. As a result of this, an IR image and a BW image, whichhave a substantially same viewpoint can be acquired, and thereforeappropriate second depth information can be acquired.

Of course, second depth estimator 111 b may use a process other than theguided filter. For example, second depth estimator 111 b may use aprocess such as bilateral filter (Non Patent Literature: C. Tomasi, R.Manduchi, “Bilateral filtering for gray and color images”, IEEEInternational Conference on Computer Vision (ICCV), pp. 839-846, 1998),or Mutual-Structure for Joint Filtering (Non Patent Literature: XiaoyongShen, Chao Zhou, Li Xu and Jiaya Jia, “Mutual-Structure for JointFiltering”, IEEE International Conference on Computer Vision (ICCV),2015) may be used.

As described above, in the present embodiment, the first depthinformation is used for a region which is discriminated that dust andthe like have not occurred (that is, a non-dust region), and the seconddepth information is used for the region where dust and the like haveoccurred (that is, a dust region). As a result of this, even if dust andthe like have occurred in the IR image, more accurate depth informationcan be acquired.

Variation

In the above embodiment, a filter such as a guided filter is used togenerate the second depth information, but the second depth informationmay be generated by using a learning model.

For example, like Non Patent Literature (Shuran Song, Fisher Yu, AndyZeng, Angel X. Chang, Manolis Savva and Thomas Funkhouser, “SemanticScene Completion from a Single Depth Image”, IEEE Conference on ComputerVision and Pattern Recognition (CVPR), pp. 190-198, 2017), Deep Learningwhich is learning process may be used. That is, the learning model maybe trained such that when the BW image and the first depth informationare inputted, the second depth information is outputted. The above NonPatent Literature proposes a network which, when inputted with depthinformation including a missing region and a color image, interpolatesthe missing region of the depth information. Second depth estimator 111b in this variation gives an IR image, a BW image, and first depthinformation to a network (that is, a learning model) similar to that ofthe Non Patent Literature, and further gives a dust region detected bydust detector 112 as a mask image of the missing region. This makes itpossible to acquire more accurate second depth information from thenetwork.

FIG. 26 is a block diagram illustrating an example of the functionalstructure of depth acquisition device 1 according to this variation.

Depth acquisition device 1 in this variation includes each componentshown in FIG. 8, and further includes learning model 104 including, forexample, a neural network.

Second depth estimator 111 b inputs three types of data: an IR image, aBW image, and first depth information, into learning model 104, andgenerates second depth information by using a dust region as a maskregion to be corrected.

In the training of learning model 104, in addition to the IR image, theBW image, and the first depth information, a correct depth image isprepared in advance. Next, the IR image, the BW image, the first depthinformation, and the mask image that specifies a dust region are givento learning model 104 as input. Then, learning model 104 is trained sothat the output from learning model 104 with respect to the inputmatches the correct depth image. It should be noted that at the time oftraining, mask images are randomly given. Second depth estimator 111 buses learning model 104 that has been trained in advance in this way.That is, second depth estimator 111 b can acquire second depthinformation outputted from learning model 104 by inputting the IR image,the BW image, the first depth information, and the mask image thatspecifies the dust region to learning model 104.

In this way, in this variation, second depth estimator 111 b estimatesthe depth information indicating the depth at each position in the IRimage, and corrects the depth at each position in the dust area asindicated by its depth information by inputting the IR image, the BWimage, the dust region, and the depth information thereof to thelearning model. Therefore, if the learning model is trained in advancesuch that a correct depth at each position in the dust region isoutputted for the inputs of the IR image, the BW image, the dust region,and the depth information, it is possible to appropriately correct thedepth information estimated from the IR image. That is, it is possibleto appropriately correct the depth at each position in the dust regionindicated by the depth information.

As described above, second depth estimator 111 b may use Deep Learning.In that case, it is not necessary to directly output the dust region,and the second depth information may be directly generated by DeepLearning.

FIG. 27 is a block diagram illustrating another example of a functionalstructure of depth acquisition device 1 according to this variation.

Depth acquisition device 1 in this variation does not include dustdetector 112, higher-luminance-region detector 116, and FOE detector117, and include components other than these among the components shownin FIG. 26.

In the training of learning model 104, a correct depth image is preparedin advance in addition to the IR image, the BW image, and the firstdepth information, as in the example shown in FIG. 26. Next, the IRimage, the BW image, and the first depth information are given tolearning model 104 as input. Then, learning model 104 is trained suchthat the output from learning model 104 with respect to the inputmatches the correct depth image. As learning model 104, a VGG-16 networkto which Skip connection is added may be used as in Non PatentLiterature (Caner Hazirbas, Laura Leal-Taixe and Daniel Cremers.Hazirbas, “Deep Depth From Focus”, In ArXiv preprint arXiv, 1704.01085,2017). The number of channels of learning model 104 is changed such thatthe IR image, the BW image, and the first depth information are given asinputs to that learning model 104. By using learning model 104 that hasbeen trained in advance in this way, second depth estimator 111 b caneasily obtain second depth information from that learning model 104 byinputting the IR image, the BW image, and the first depth information tolearning model 104.

That is, depth acquisition device 1 shown in FIG. 27 includes a memoryand processor 110. It should be noted that the memory, though not shownin FIG. 27, may be provided in depth acquisition device 1 as shown inFIG. 5. Processor 110 acquires timing information indicating the timingat which light source 101 irradiates the subject with infrared light.Next, processor 110 acquires an IR image, which is obtained by imaging ascene including a subject with infrared light according to the timingindicated by the timing information, and is retained in a memory. Next,processor 110 acquires a BW image which is retained in a memory andobtained by imaging of a substantially same scene as that of the IRimage, the imaging being performed with visible light at a substantiallysame viewpoint and imaging time as those of the IR image. Then, firstdepth estimator 111 a of processor 110 estimates the depth informationindicating the depth at each position in the IR image. By inputting theIR image, the BW image, and the depth information to learning model 104,second depth estimator 111 b corrects the depth at each position in thedust region of the IR image indicated by the depth information.

Therefore, if learning model 104 is trained in advance such that acorrect depth at each position in the dust region of the IR image isoutputted for the inputs of the IR image, the BW image, and the depthinformation, it is possible to appropriately correct the depthinformation estimated from the IR image. That is, it is possible toappropriately correct the depth at each position in the dust regionindicated by the depth information without detecting the dust region.

As described so far, in depth acquisition device 1 in the presentembodiment and its variation, even when there is a dust region in the IRimage, it is possible to acquire appropriate depth at each position inthe dust region by using the image of the corresponding region of the BWimage.

It is noted that in the above-described each embodiment, each componentmay be constituted by dedicated hardware or may be implemented byexecuting a software program suitable for each component. Each componentmay be implemented by a program executing unit, such as a CPU or aprocessor, reading and executing a software program recorded on arecording medium such as a hard disk or a semiconductor memory. Here,the software that implements the depth acquisition device and the likeof the above-described embodiments and variations causes the computer toexecute each step included in the flowchart of any of FIGS. 6, 7, and 21to 25.

Although the depth acquisition device according to one or more aspectshas been described above based on the embodiment and its variations, thepresent disclosure is not limited to this embodiment and its variations.Embodiments in which various modifications that can be conceived bythose skilled in the art are applied to the present embodiment and itsvariations, and embodiments which are constructed by combiningcomponents in the present embodiment and its variations may be includedwithin the scope of the present disclosure as long as they do not departfrom the gist of the present disclosure.

For example, in the above-described embodiment and its variations, dustdetector 112 detects a dust region, but it may detect a region otherthan the dust region as long as an object is projected as noise likedust on the region. For example, dust detector 112 may detect a rainregion on which raindrops are projected or a snow region on whichsnowflakes are projected. In an environment where it is raining lightly,if the size of raindrop is sufficiently smaller than the resolution ofthe BW image acquired by BW camera 103 or BW image acquirer 115, theraindrops will not be reflected in the BW image. However, in an IR imageacquired by IR camera 102 or IR image acquirer 114, the infrared lightfrom light source 101 is reflected on raindrops and observed as highluminance. Therefore, in the first depth information or the first depthimage generated by first depth estimator 111 a, the depth of the rainregion will become inappropriate. Similarly, in a snowfall environment,if the size of snowflake is sufficiently smaller than the resolution ofthe BW image acquired by BW camera 103 or BW image acquirer 115, thesnowflakes will not be reflected in the BW image. However, in the IRimage acquired by IR camera 102 or IR image acquirer 114, infrared lightfrom light source 101 is reflected by the snowflakes and observed ashigh luminance. Therefore, in the first depth information or the firstdepth image generated by first depth estimator 111 a, the depth of thesnow region will become inappropriate. Therefore, dust detector 112detects a rain region or a snow region by the same method as that forthe detection of a dust region. Consequently, second depth estimator 111b uses a BW image, first depth information, and a region detected bydust detector 112 (that is, a rain region or a snow region) to generatesecond depth information. As a result of this, it is possible to acquiresecond depth information that is not affected by rain or snow. It isnoted that the dust in the present disclosure includes solid particlessuch as dust, but is not limited to solid particles and may includeliquid particles. For example, the dust in the present disclosure mayinclude at least one type of dust, raindrops, and snowflakes.

It should also be noted that all or a part of the units and the devicesaccording to the present disclosure or all or a part of the functionalblocks in the block diagrams of FIGS. 1, 4, 5, 8, 26, and 27 may beimplemented into one or more electronic circuits including asemiconductor device, a semiconductor Integrated Circuit (IC), or aLarge Scale Integration (LSI). The LSI or the IC may be integrated intoa single chip, or may be a combination of multiple chips. For example,the functional blocks except the storage element may be integrated intoa single chip. Here, the LSI or the IC may be referred differentlydepending on the degree of integration, and may also be referred to as asystem LSI, a Very Large Scale Integration (VLSI), or an Ultra LargeScale Integration (ULSI). A Field Programmable Gate Array (FPGA) whichis programmable after manufacturing an LSI or a reconfigurable logicdevice capable of reconfiguring the connections or circuit segmentationin the LSI circuit may be used for the same purpose.

Furthermore, functions or operations of all or a part of the units, thedevices, or a part of the devices may be realized by executing asoftware program. In this case, the software program is recorded on oneor more nontransitory recording mediums such as a Read Only Memory(ROM), an optical disk, or a hard disk drive. When the software programis executed by a processor, the software program causes the processorand its peripheral devices to execute specific functions in the softwareprogram. The system or the device may include such one or morenon-transitory recording medium on which the software program isrecorded, a processor, and necessary hardware devices such as aninterface.

INDUSTRIAL APPLICABILITY

The present disclosure is applicable to depth acquisition devices thatacquire a depth of an image generated by imaging, for example,applicable to in-vehicle devices and the like.

What is claimed is:
 1. A depth acquisition device, comprising: a memory;and a processor, wherein the processor performs: acquiring timinginformation indicating a timing at which a light source irradiates asubject with infrared light; acquiring an infrared light image stored inthe memory, the infrared light image being generated by imaging a sceneincluding the subject with the infrared light according to the timingindicated by the timing information; acquiring a visible light imagestored in the memory, the visible light image being generated by imaginga substantially same scene as the scene of the infrared light image,with visible light from a substantially same viewpoint as a viewpoint ofthe imaging the infrared light image at a substantially same time as animaging time of imaging the infrared light image; detecting a dustregion showing dust from the infrared light image; and estimating adepth of the dust region based on the infrared light image, the visiblelight image, and the dust region.
 2. The depth acquisition deviceaccording to claim 1, wherein in the estimating of the depth of the dustregion, the processor performs: estimating first depth informationindicating a depth at each position in the infrared light image;estimating second depth information indicating a corrected depth at eachposition in the dust region, the corrected depth being obtained bycorrecting, based on the visible light image, a depth at each positionin the dust region which is indicated in the first depth information;and generating third depth information indicating (i) a depth at eachposition in a region other than the dust region in the infrared lightimage which is indicated in the first depth information and (ii) a depthat each position in the dust region in the infrared light image which isindicated in the second depth information.
 3. The depth acquisitiondevice according to claim 1, wherein in the detecting of the dustregion, the processor performs detecting, as the dust region, a higherluminance region having a luminance not less than a first threshold inthe infrared light image, when the higher luminance region satisfies afirst condition.
 4. The depth acquisition device according to claim 3,wherein the first condition is that a center of gravity of the higherluminance region is located on a straight or an arc, the straight lineor the arc intersecting (i) a center of gravity of each of at least twohigher luminance regions other than the higher luminance region in theinfrared light image and (ii) a Focus of Expansion (FOE) of one of theinfrared light image and the visible light image.
 5. The depthacquisition device according to claim 3, wherein the first condition isthat one of a principle axis of the higher luminance region and a lineextending from the principle axis intersects a Focus of Expansion (FOE)of one of the infrared light image and the visible light image.
 6. Thedepth acquisition device according to claim 3, wherein in the detectingof the dust region, the processor performs detecting the higherluminance region as the dust region, when the1 higher luminance regionfurther satisfies a second condition.
 7. The depth acquisition deviceaccording to claim 6, wherein the second condition is that a luminanceof a position in the visible light image is less than a secondthreshold, the position in the visible light image corresponding to acenter of gravity of the higher luminance region in the infrared lightimage.
 8. The depth acquisition device according to claim 6, wherein thesecond condition is that a correlation coefficient between (i) aluminance in the higher luminance region in the infrared light image and(ii) a luminance in a region in the visible light image is less than athird threshold, the region in the visible light image corresponding tothe higher luminance region.
 9. The depth acquisition device accordingto claim 1, wherein in the estimating of the depth of the dust region,the processor performs: estimating depth information indicating a depthat each position in the infrared light image; and correcting a depth ateach position in the dust region which is indicated in the depthinformation, by inputting the infrared light image, the visible lightimage, the dust region, and the depth information into a learning model.10. A depth acquisition device, comprising: a memory; and a processor,wherein the processor performs: acquiring timing information indicatinga timing at which a light source irradiates a subject with infraredlight; acquiring an infrared light image stored in the memory, theinfrared light image being generated by imaging a scene including thesubject with the infrared light according to the timing indicated by thetiming information; acquiring a visible light image stored in thememory, the visible light image being generated by imaging asubstantially same scene as the scene of the infrared light image, withvisible light from a substantially same viewpoint as a viewpoint of theimaging the infrared light image at a substantially same time as animaging time of imaging the infrared light image; estimating depthinformation indicating a depth at each position in the infrared lightimage; and correcting a depth at each position in a dust region showingdust in the infrared light image by inputting the infrared light image,the visible light image, and the depth information into a learningmodel, the depth at each position in the dust region being indicated inthe depth information.
 11. A depth acquisition device, comprising: amemory; and a processor, wherein the processor performs: acquiring aninfrared light image stored in the memory, the infrared light imagebeing generated by imaging with infrared light; acquiring a visiblelight image stored in the memory, the visible light image beinggenerated by imaging with visible light from a substantially sameviewpoint as a viewpoint of the imaging the infrared light image at asubstantially same time as an imaging time of imaging the infrared lightimage; detecting, as a dust region, a region showing dust from theinfrared light image; and estimating a depth of the dust region based onthe visible light image.
 12. A depth acquisition method, comprising:acquiring timing information indicating a timing at which a light sourceirradiates a subject with infrared light; acquiring an infrared lightimage stored in a memory, the infrared light image being generated byimaging a scene including the subject with the infrared light accordingto the timing indicated by the timing information; acquiring a visiblelight image stored in the memory, the visible light image beinggenerated by imaging a substantially same scene as the scene of theinfrared light image, with visible light from a substantially sameviewpoint as a viewpoint of the imaging the infrared light image at asubstantially same time as an imaging time of imaging the infrared lightimage; detecting a dust region showing dust from the infrared lightimage; and estimating a depth of the dust region based on the infraredlight image, the visible light image, and the dust region.
 13. A depthacquisition method, comprising: acquiring timing information indicatinga timing at which a light source irradiates a subject with infraredlight; acquiring an infrared light image stored in a memory, theinfrared light image being generated by imaging a scene including thesubject with the infrared light according to the timing indicated by thetiming information; acquiring a visible light image stored in thememory, the visible light image being generated by imaging asubstantially same scene as the scene of the infrared light image, withvisible light from a substantially same viewpoint as a viewpoint of theimaging the infrared light image at a substantially same time as animaging time of imaging the infrared light image; estimating depthinformation indicating a depth at each position in the infrared lightimage; and correcting a depth at each position in a dust region showingdust in the infrared light image by inputting the infrared light image,the visible light image, and the depth information into a learningmodel, the depth at each position in the dust region being indicated inthe depth information.
 14. A depth acquisition method using a depthacquisition device, the depth acquisition device including a memory anda processor, the processor performing: acquiring an infrared light imagestored in the memory, the infrared light image being generated byimaging with infrared light; acquiring a visible light image stored inthe memory, the visible light image being generated by imaging withvisible light from a substantially same viewpoint as a viewpoint of theimaging the infrared light image at a substantially same time as animaging time of imaging the infrared light image; detecting, as a dustregion, a region showing dust from the infrared light image; andestimating a depth of the dust region based on the visible light image.